Digital processing method and system for determination of object occlusion in an image sequence

ABSTRACT

A method and system for occlusion region detection and measurement between a pair of images are disclosed. A processing device receives a first image and a second image. The processing device estimates a field of motion vectors between the first image and the second image. The processing device motion compensates the first image toward the second image to obtain a motion-compensated image. The processing device compares a plurality of pixel values of the motion-compensated image to a plurality of pixels of the first image to estimate an error field. The processing device inputs the error field to a weighted error cost function to obtain an initial occlusion map. The processing device regularizes the initial occlusion map to obtain a regularized occlusion map.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit of U.S.utility patent application Ser. No. 14/065,704 filed Oct. 29, 2013,which claims the benefit of U.S. provisional patent application No.61/769,311 filed Feb. 26, 2013, the disclosures of which areincorporated herein by reference in their entirety.

TECHNICAL FIELD

The present invention relates generally to digital image processing, andmore particularly, to a method and system for automatic determination ofthe presence, location, and type of occlusion regions between a pair ofimages.

BACKGROUND

Determining an optical flow or motion vector field between two images,particularly for sequences of video frames and/or fields, is frequentlyencountered in many high-value video processing tasks such as coding,frame rate conversion, noise reduction, etc. Conventional methods forcalculating optical flow encounter several stumbling blocks—manysolutions of which are described in U.S. Pat. No. 8,355,534(hereinafter, “the '534 patent”), incorporated herein by reference inits entirety. As taught in the '534 patent, object occlusion presents achallenge for any motion estimation system, such as an optical flowestimation system.

FIG. 1 shows an example of an image pair 100 a, 100 b, with background105 and foreground 110, where a foreground object 115 is in motion andwhich shows an occlusion region 120 and a disocclusion region 125. Whenthe foreground object 115 is in motion in a video sequence, backgroundpixels of the image 100 b in the forward-motion direction are hidden(known herein as occlusion or the occlusion region 120) while backgroundpixels of the image 100 b behind the motion are revealed (known hereinas disocclusion or the disocclusion region 125). In the occluded areasof an image, there is no definite motion attributable to the background;concomitantly, there is no definite motion attributable to theforeground object in disoccluded regions of the image. These two typesof areas within a pair of images (collectively known herein as occlusionregions) are very problematic for motion estimation in general, and formany optical flow systems in particular, because erroneous motion vectorvalues in these regions tend to propagate into non-occlusion regions,adversely affecting the overall accuracy of the optical flow estimation.Determination of occlusion regions has many benefits for otherhigh-value video analysis tasks in addition to improvement of opticalflow and motion estimation, such as disparity and depth estimation,image segmentation, object identification, and 3D conversion andprojection.

The detection of occlusion has received much attention in the context ofmotion estimation, depth estimation and image/video segmentation.Occlusion can be estimated or computed explicitly or implicitly.Occlusion boundaries themselves provide strong cues for 3D scenereconstruction. Methods as described in A. Saxena, M. Sun, and A. Y. Ng,“Make 3D: Learning 3D Scene structure form a Single Image,” PAMI, 31:824-840, 2009, and in D. Hoiem, A. A. Efros, and A. Hebert, “RecoveringOcclusion Boundaries from an Image,” International Journal on ComputerVision, pages 1-19, 2010, propose to find occlusion boundaries using asingle frame by over-segmentation and supervised-learning. With nomotion information, occlusion boundary detection is an inherentlyambiguous problem. Other methods attempt to layer input video intoflexible sprites to infer occluded pixels/regions (see e.g., N. Jojicand B. J. Frey, “Learning Flexible Sprites in Video layers,” in CVPR,2001). Layered methods provide realistic modeling of occlusionboundaries, but these methods need to have continuous regions, relativeorder of surfaces, and predetermined motion. The method described inSun, D., Sudderth, E. B., Black, M. J., “Layered image motion withexplicit occlusions, temporal consistency, and depth ordering,” in:Advances in Neural Information Processing Systems, pp. 2226-2234 (2010),explicitly models occlusion and the results obtained are relativelyaccurate, but the method possesses a huge computational load. Findingocclusion regions represents a common problem in multi-view 3Dprojection and display methods. The most recent researched methods inthis area are still prone to gross errors when the background orforeground underlying pixel data in these regions is homogeneous or haveno texture information.

In Alvarez, et al., “Symmetrical dense optical flow estimation withocclusions detection,” International Journal of Computer Vision 75(3),371-385 (2007), (hereinafter, Alvarez), passing interest is focused onthe role of the diffusion tensor and subsequent eigenvalue analysis, butthis is only used to analyze the forward and backward symmetry of theoptical flow solution, and not used to directly improve the accuracy ofeither the optical flow computation nor the occlusion computation.

Ince, S., Konrad, J., “Occlusion-aware optical flow estimation,” IEEETrans. Image Processing 17(8), 1443-1451 (2008), (hereinafter, “Ince”),discloses a method and systems for joint determination of optical flowand occlusion, but the systems are coupled and this method is notapplicable for coupling to a non-optical-flow motion estimation system,such as block matching. Further, Ince ignores the notion of either adiffusion tensor or structure tensor of the images in order to improverobustness.

Motion cues are very important for identifying occlusion regions andboundaries. As described above, the objective of any motion estimationis to compute a flow field that represents the motion of points in twoconsecutive frames, and the most accurate motion estimation techniquesshould be able to handle occlusions. Some occlusion detection work basedon motion as described in Alvarez and Ince, jointly estimates backwardand forward motion and marks inconsistent pixels as occluded regions. Insuch circumstances, occlusion is detected implicitly and the occlusiondetection is coupled with the motion estimation method itself. Thesemethods encounter problems within highly textured imagery areas and donot succeed with large displacements or occlusion regions.

Xiao, et al., “Bilateral Filtering-Based Optical Flow Estimation withOcclusion Detection,” Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV2006, Part I. LNCS, vol. 3951, pp. 211-224, Springer, Heidelberg (2006)discloses another joint method for computing optical flow and occlusion,but its two computations are closely coupled into a joint regularizationframework. Further, this method requires multiple iterations forconvergence of the disclosed regularization function and is thereforenot suitable for real-time computation for contemporaneous videoresolutions such as 1080 and 4K.

Even the best conventional motion estimation methods with coupledocclusion detection systems suffer from two primary disadvantages.First, these methods are too computationally complex for real-timeprocessing. Second, the occlusion region maps they produce areinherently noisy. Pixels marked as occlusions may frequently befalse-positives or false-negatives, rendering their usage in subsequentvideo processing and analysis tasks challenging or impossible.

Accordingly, there is a need for an accurate, precise, low-computationalcomplexity occlusion estimation system and method that in conjunctionwith a motion estimation system, increases the robustness and accuracyof such a system in the presence of large motions and resulting largeocclusion regions.

BRIEF SUMMARY OF THE INVENTION

The above-described problems are addressed and a technical solution isachieved in the art by providing a method and system for occlusionregion detection and measurement between a pair of images. A processingdevice receives a first image and a second image. The processing deviceestimates a field of motion vectors between the first image and thesecond image. The processing device motion compensates the first imagetoward the second image to obtain a motion-compensated image. Theprocessing device compares a plurality of pixel values of themotion-compensated image to a plurality of pixels of the first image toestimate an error field. The processing device inputs the error field toa weighted error cost function to obtain an initial occlusion map. Theprocessing device regularizes the initial occlusion map to obtain aregularized occlusion map.

In one example, regularizing may further comprise obtaining aregularized error field. In one example, comparing and regularizing maybe repeated until a value based on at least one of the regularizedocclusion map or the regularized error field is below a threshold value.In one example, motion compensating the first image toward the secondimage comprises image warping the field of motion vectors from the firstimage toward the second image. In one example, the initial occlusion mapand the regularized occlusion map are each based on a weighted errorcost function. The weighted error cost function may be at least one of asum-of-square differences measure, a locally scaled sum-of-squaredifferences measure, a normalized cross-correlation measure, or azero-mean normalized cross-correlation measure. The weighted error costfunction may be based on a local weighting over a local region ofsupport. In one example, the local weighting over a local region ofsupport may be based on an eigensystem analysis of the local structuretensor of the motion-compensated image. In another example, the localweighting over a local region of support is a gradient-energy weightingover the local region of support. The gradient-energy weighting over alocal region of support may be a sum of statistical variance or localcontrast over the local region of support.

In one example, regularizing the occlusion map to obtain a regularizedocclusion map may comprise applying a multi-sigma regularization to theocclusion map. Applying a multi-sigma regularization to the occlusionmap may comprise applying a 4-factor sigma filter to the occlusion map.Input weights for the multi-factor sigma filter may comprise an initialcoarse occlusion field estimate and, between the first image and thesecond image one or more of similarities of color value or luminance,similarities of circular values of motion vector directions, orsimilarities of motion vector magnitudes. The multi-factor sigma filtermay incorporate one or more weights, such as depth, or discontinuitiesof a range-to-target field.

Examples of the present disclosure provide a method and system fordetecting and characterizing occlusion regions without any assumptionsthat depend on scene types, motion types, or supervised learningdatasets. Examples of the present disclosure provide an accurate andprecise occlusion region map. The occlusion detection is decoupled fromthe motion estimation itself, providing for flexible addition to anysuitable optical flow or motion estimation system or method.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of an image pair with background and foreground,where the foreground object is in motion and shows occlusion anddisocclusion regions.

FIG. 2 is a block diagram of an example computing system for detectingone or more occlusion regions in an image sequence, in which examples ofthe present disclosure may operate.

FIG. 3 shows an exemplary occlusion field/map.

FIG. 4 is a flow diagram illustrating an example of a method fordetecting occlusion regions and/or disocclusion regions in a sequence ofimages using the computing system of FIG. 2.

FIG. 5 is a block diagram of an example data flow between modules thatimplement the method of FIG. 4.

FIG. 6 is a block diagram of an example data flow between modules thatimplement a weighted distance field module of FIG. 5.

FIG. 7 is a block diagram of an example data flow through a 4-factorsigma filter employed in a regularization module of FIG. 5.

FIG. 8 illustrates a diagrammatic representation of a machine in theexample form of a computer system within which a set of instructions,for causing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

DETAILED DESCRIPTION

Motion estimation has been extensively explored in the related art and adetermination of accurate motion vectors between images is still achallenging problem. Several solutions have historically been used formotion estimation, such as simple block matching, hierarchical blockmatching, and optical flow estimation. In order to estimate an accuratemotion vector field, the occlusion problem needs to be explicitlyconfronted. Occlusion detection is also very important in estimating adisparity map for stereo matching. Occlusion marking has independentapplications beyond motion estimation and disparity map estimation, suchas in video surveillance object sorting/object removal and re-projectionof multi-view video and imagery. Also, marking occlusions is very usefulfor image segmentation, motion segmentation, image in-painting, anddisparity in-painting.

Occlusion occurs under one of the following conditions: a camera orcapture system is zoomed in/out, a new object appears in-frame, or oldobjects disappear from frame, and moving foreground objects revealbackground pixels. For example, in stereoscopic image pairs, two imagesare captured from different angles and some pixels exist in only oneview. As used herein, these pixels are known as occluded pixels.Similarly, in video surveillance, a person moving away from a camera ora new person appearing in front of the camera introduces occlusion.

FIG. 2 is a block diagram of an example computing system 200 fordetecting one or more occlusion regions in an image sequence, in whichexamples of the present disclosure may operate. By way of non-limitingexample, the computing system 200 receives data from one or more datasources 205, such as a video camera or a still camera or an on-linestorage device or transmission medium. The computing system 200 may alsoinclude a digital video capture system 210 and a computing platform 215.The digital video capturing system 210 processes streams of digitalvideo, or converts analog video to digital video, to a form which can beprocessed by the computing platform 215 as data source 205. Thecomputing platform 215 comprises a host system 220 which may comprise,for example, a processing device 225, such as one or more centralprocessing units 230 a-230 n. The processing device 225 is coupled to ahost memory 235.

The processing device may further implement a graphics processing unit240 (GPU). It will be appreciated by those skilled in the art that otherco-processor architectures may be utilized besides GPUs, such as, butnot limited to, DSPs, FPGAs, or ASICs, or adjunct fixed-functionfeatures of the processing device 225 itself. It will further beappreciated by those skilled in the art that the GPU 240 may becollocated on the same physical chip or logical device as the centralprocessing units 230 a-230 n, also known as an “APU”, such as found onmobile phones and tablets. Separate GPU and CPU functions may be foundon computer server systems where the GPU is a physical expansion card,and personal computer systems and laptops. The GPU 240 may comprise aGPU memory 237. It will be appreciated by those skilled in the art thatthe host memory 235 and GPU memory 237 may also be collocated on thesame physical chip(s) or logical device, such as on an APU.

The processing device 225 is configured to implement an occlusion mapgenerator 245 for detecting occlusion regions and/or disocclusionregions in a sequence of images. The occlusion map generator 245 may beconfigured to receive data (e.g., a first image and a second image) fromthe data source 205, and to receive an image data buffer 250, which istransferred to the GPU memory 237 as image buffer 255. In one example,the processing device 225 may implement the occlusion map generator 245as a component of the GPU 240. The occlusion map generator 245 isconfigured to obtain a regularized occlusion map from the image buffer255 as shown in FIG. 3. As used herein, the totality of occluded regionsfor a given image is referred to as an occlusion map. In one example,the regularized occlusion map may be displayed on a display 270. Inanother example, the occlusion map generator 245 may transmit theregularized occlusion map to one or more downstream devices 290 directlyor through a network 295.

FIG. 4 is a flow diagram illustrating an example of a method 400 fordetecting occlusion regions and/or disocclusion regions in a sequence ofimages. The method 400 may be performed by a computer system 200 of FIG.2 and may comprise hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device), or a combination thereof. In one example, themethod 400 is performed by the occlusion map generator 245 of thecomputing system 200 of FIG. 2.

As shown in FIG. 4, to permit the computing system 200 to generate aregularized occlusion map from a sequence of images, at block 410, theocclusion map generator 245 estimates a field of motion vectors betweena first image and a second image based on the received data from theimage buffer 255. At block 420, the occlusion map generator 245 operateson the field of motion vectors to motion compensate the first imagetoward the second image to obtain a motion-compensated image. In oneexample, when the occlusion map generator 245 motion compensates thefirst image toward the second image, the occlusion map generator 245image warps the field of motion vectors from the first image toward thesecond image.

At block 430, the occlusion map generator 245 compares a plurality ofpixel values of the motion-compensated image to a plurality of pixels ofthe first image to estimate an error field. At block 440, the occlusionmap generator 245 inputs the error field to a weighted error costfunction to obtain an initial occlusion map. In one example, theweighted error cost function may be at least one of a sum-of-squaredifferences measure, a locally scaled sum-of-square differences measure,a normalized cross-correlation measure, or a zero-mean normalizedcross-correlation measure. In one example, the weighted error costfunction may be based on a local weighting over a local region ofsupport. The local weighting over a local region of support may be basedon an eigensystem analysis of the local structure tensor of themotion-compensated image. In one example, the local weighting over alocal region of support may be a gradient-energy weighting over thelocal region of support. In an example, the gradient-energy weightingover a local region of support may be a sum of statistical variance orlocal contrast over the local region of support.

At block 450, the occlusion map generator 245 regularizes the initialocclusion map to obtain a regularized occlusion map. In one example,when the occlusion map generator 245 regularizes the initial occlusionmap, the occlusion map generator 245 may further regularize the errorfield.

In one example, regularizing the occlusion map to obtain a regularizedocclusion map may comprise applying a multi-sigma regularization to theocclusion map and the error field. In one example, applying amulti-sigma regularization to the occlusion map and error field maycomprise applying a 4-factor sigma filter to the occlusion map.

At block 460, if a value based on at least one of the regularizedocclusion map or the regularized error field is above a threshold value,the occlusion map generator 245 repeats the comparing and theregularizing steps (i.e., repeating steps 430-460); otherwise,processing terminates.

Let f(x,y,t) be the current frame denotes frame and f(x,y, t−1) be theprevious frame. The relationship between f(x,y,t) and f(x,y,t−1) can bedenoted as Eq. 1:

$\begin{matrix}{{\hat{f}( {x,y,t} )} \approx \{ \begin{matrix}{W_{{t - 1} = {> t}}( {f( {x,y,{t - 1}} )} } \\{o( {x,y,t} )}\end{matrix} } & {{Eq}.\mspace{14mu} 1}\end{matrix}$where W_(t->(t-1)) denotes the motion-compensated mapping of framef(x,y,t−1) to frame f(x,y,t).

Thus the operator gives a per-pixel mapping between two frames and thisis applicable to any motion model and estimating disparity map. Equation(1) holds true for everywhere in the video frame except for the occludedregions o(x,y,t−1). The totality of occluded regions for a given imageis referred to as the occlusion map. This map represents a gray scaleimage mask, or alternatively a 2-dimensional matrix of positive values,that in ideal circumstances accurately identifies each pixel as beingoccluded, where middle range values can either represent variations inconfidence, or alternatively degree of transparency of the occludingobject(s). In the present disclosure, the problems addressed are offinding the occluded areas in a sequence of images and to performregularization of the resulting occlusion map to attain temporalstability and to prevent recursive error propagation.

FIG. 5 is a block diagram of one example of data flow through a sequenceof modules 500 that comprise the occlusion map generator 245 of FIG. 3.A motion estimation module 508 estimates motion vectors of a motionvector field 502 from a first frame 504 (e.g., a previous frame 504)f(x,y,t−1) to a second frame 506 (e.g., an original frame 506) framef(x,y,t). The motion estimation module 508 may implement a motionestimate method which can be, for example, a sub-pixel hierarchicalblock-based method, optical flow, or recursive disparity estimation forstereoscopic pairs. Based on the motion vectors mv_(u)(x,y, t−1) andmv_(v)(x,y, t−1) calculated, a motion compensated warping module 510 mayapply a motion compensation warping function W_(t->(t-1)) to the motionvector field 502 to obtain a motion warped first frame 512 (e.g., amotion warped previous frame 512). In an example, the functionW_(t->(t-1)) can be expanded as Eq. 2,(x+mv _(u)(x,y,t−1),y+mv _(v)(x,y,t−1),t)≈(x,y,t−1)  Eq. 2

It will be appreciated by those skilled in the art that any number ofmotion compensation regimes may be employed, and the above example isprovided for clarity of explanation. The occluded regions will not havetrue motion vectors since no information was available for the motionestimation system. An ideal motion estimation system may populate theseareas with zero magnitude motion vectors or at least signal a lack ofconfidence in their accuracy by an out-of-band method such as bycommunicating a confidence map.

It is important to distinguish the warping method described with typicalmotion compensation regimes. In most motion compensation systems, a“gather” method is applied, such that:(x,y,t)≈(x+mv _(u)(x,y,t−1),y+mv _(v)(x,y,t−1),t−1)  Eq. 3

While the regime as in Eq. 3 ensures that every destination pixel in thecompensation will be visited, ensuring a dense image, it also ignoresocclusion. By comparison, the warping method of Eq. 2 can be describedas a “scatter” method whereby every pixel of the compensated image isnot guaranteed to be visited, or filled in. By pre-filling thecompensated image buffer with a signal value, unvisited regions in thescatter-based warping operation are left with this signal valueundisturbed. This, in turn, forms the starting point of the occlusionmap 514.

Next, since the second or original (true) f(x,y,t) frame 506 is known, aweighting function block 516 may apply a weighting function to themotion warped first (e.g., previous) frame 512 to obtain an weightingfield 513 which informs subsequent processing steps of the weight to beapplied to error cost function analysis for each pixel. In an example,the weighting field 513 may be stored in a separate data buffer or inthe alpha channel or fourth channel of the motion warped first (e.g.,previous) frame 512. The weighting function may comprise a simpleidentity function, or something more complex such as Eigensystemanalysis of the local structure tensor.

A weighted error cost function block 518 may apply a weighted error costfunction using the weights supplied by the weighting field 513. Errorpixels from the motion warped first (e.g., previous) frame 512 can becalculated from the weighted error cost function and thereby theoccluded areas can be further marked while avoiding the areas alreadymarked.

Nevertheless, special consideration is needed when choosing error costfunctions to estimate error, since simple per-pixel differences (knowncommonly as Sum-of-Absolute-Differences) or un-weighted sum of squareddifferences may mark false negatives for homogenous regions and objectswith low texture and false positives in image regions with strongcontrast and edges. The following correlation based similarity measures(see e.g., Nuno Roma, José Santos-Victor, José Tomé, “A ComparativeAnalysis Of Cross-Correlation Matching Algorithms Using a PyramidalResolution Approach,” 2002) for estimating the error pixels for eachpixel in a frame are instructive for the purpose of the preferredembodiment, but are not a limiting example:

Sum of Squared Differences (SSD),

$\begin{matrix}{{e( {x,y,t} )} = {\sum\limits_{{({i,j})} \in R}( {{I( {i,j,t} )} - {\hat{I}( {{x + i},{y + j},t} )}} )^{2}}} & {{Eq}.\mspace{14mu} 4}\end{matrix}$Locally scaled Sum of Squared Differences (LSSD),

$\begin{matrix}{{e( {x,y,t} )} = {\sum\limits_{{({i,j})} \in R}{{{I( {i,j,t} )} - {\frac{\overset{\_}{I( {i,j,t} )}}{\overset{\_}{\hat{I}( {x,y,t} )}}{\hat{I}( {{x + i},{y + j},t} )}}}}}} & {{Eq}.\mspace{14mu} 5}\end{matrix}$Normalized Cross Correlation (NCC), and

$\begin{matrix}{{e( {x,y,t} )} = \frac{\sum\limits_{{({i,j})} \in R}( {{I( {i,j,t} )} - {\hat{I}( {{x + i},{y + j},t} )}} )^{2}}{\sqrt{\sum\limits_{{({i,j})} \in R}{( {I( {i,j,t} )} )^{2} \cdot {\sum\limits_{{({i,j})} \in R}( {I( {{x + i},{y + j},t} )} )^{2}}}}}} & {{Eq}.\mspace{14mu} 6}\end{matrix}$Zero-Mean Normalized Cross Correlation (ZNCC):

$\begin{matrix}{{e( {x,y,t} )} = \frac{\sum\limits_{{({i,j})} \in R}{( {{I( {i,j,t} )} - \overset{\_}{I}} ) \cdot ( {{\hat{I}( {{x + i},{y + j},t} )} - \overset{\overset{\_}{\hat{}}}{I}} )}}{\sqrt{\sum\limits_{{({i,j})} \in R}{( {{I( {i,j,t} )} - \overset{\_}{I}} )^{2} \cdot {\sum\limits_{{({i,j})} \in R}( {{I( {{x + i +},{y + j},t} )} - \overset{\overset{\_}{\hat{}}}{I}} )^{2}}}}}} & {{Eq}.\mspace{14mu} 7}\end{matrix}$For the equations (4)-(7) shown above, R is the region of supportconsidered for correlation matching and selecting R as 3×3 pixels may besuitable for real-time processing, and 5×5 may be suitable for offlineprocessing. It will be appreciated by those skilled in the art that,over time, larger regions of support may be employed for real-time andoffline processing as the underlying system speed and complexityincreases. The 3×3 and 5×5 regions-of-support may be provided asexamples.

Correlation based matching metrics are very computationally expensive,but since motion vectors are already estimated from an external system,the difference metric can be estimated for a smaller region and does notrequire a search in a larger pixel region of support.

In addition to the numerical methods for determining local weightingsuch as global and local mean as disclosed in Eq. 4-7 above, aneigensystem analysis can be utilized to provide a more precise andaccurate weighting. The methods described in U.S. Pat. No. 8,355,534,incorporated herein by reference, are particularly instructive, but ofparticular note here is the use of eigenvalues of the gradient structuretensor of the local region of support to determine whether the region isan isotropic, homogeneous region, one containing significant imagetexture, or one containing a strong contrast edge. Homogeneous andisotropic region image differences would be weighted less, than forexample, those in a highly textured region, based upon such a tensoranalysis.

Optical flow motion vectors and disparity maps commonly useregularization and smoothing steps to smooth discontinuities andoutliers and further helps to stabilize the motion vector fields alongthe temporal axis in case of video. It is noted that occlusion and errorfields benefit from separate treatment in the same way with aregularization module 520, apart from the motion vector field and theimage field.

To address the problems of noise, false-positives, and false negativesof the prior art in the final resulting occlusion map 514, weightingfunction 516 and weighted error cost function 518 may include aneigensystem analysis as depicted in FIG. 6. First, a spatio-temporalgradient estimation 630 may be applied to the field of pixels for theprevious frame 610 and the current frame 620 as taught in the '534patent, which results in a two-dimensional gradient field 640, whereingradient derivatives may be estimated, for example, in Eq. 9:

$\sigma_{xx} = {\sum\limits_{n = {- 1}}^{n = 1}{\sum\limits_{m = {- 1}}^{m = 1}{\frac{\partial{D_{RGB}( {n,m} )}}{\partial x} \cdot \frac{\partial{D_{RGB}( {n,m} )}}{\partial x} \cdot {{weight}( {n,m} )}}}}$$\sigma_{xy} = {\sum\limits_{n = {- 1}}^{n = 1}{\sum\limits_{m = {- 1}}^{m = 1}{\frac{\partial{D_{RGB}( {n,m} )}}{\partial x} \cdot \frac{\partial{D_{RGB}( {n,m} )}}{\partial y} \cdot {{weight}( {n,m} )}}}}$$\sigma_{yy} = {\sum\limits_{n = {- 1}}^{n = 1}{\sum\limits_{m = {- 1}}^{m = 1}{\frac{\partial{D_{RGB}( {n,m} )}}{\partial y} \cdot \frac{\partial{D_{RGB}( {n,m} )}}{\partial y} \cdot {{weight}( {n,m} )}}}}$$\sigma_{xt} = {\sum\limits_{n = {- 1}}^{n = 1}{\sum\limits_{m = {- 1}}^{m = 1}{\frac{\partial{D_{RGB}( {n,m} )}}{\partial x} \cdot \frac{\partial{D_{RBG}( {n,m} )}}{{\partial x}{\partial t}} \cdot {{weight}( {n,m} )}}}}$$\sigma_{yt} = {\sum\limits_{n = {- 1}}^{n = 1}{\sum\limits_{m = {- 1}}^{m = 1}{\frac{\partial{D_{RGB}( {n,m} )}}{\partial y} \cdot \frac{\partial{D_{RGB}( {n,m} )}}{{\partial y}{\partial t}} \cdot {{weight}( {n,m} )}}}}$

The gradient field 640 is input into a gradient tensor analysis 650,where the gradient values are input to a tensor, and the tensor issubjected to eigensystem analysis as in Eq. 10:

${2{D\_ tensor}} = {\begin{matrix}\sigma_{xx} & \sigma_{xy} \\\sigma_{xy} & \sigma_{yy}\end{matrix}}$

The eigensystem analysis of Eq. 10 results in two Eigenvalues λ₁ and λ₂for each and every pixel, where the combination of each eigenvalueidentifies the local structure of the image surrounding any aforesaidpixel.

The Eigenvalues obtained from gradient tensor analysis 650 result inEigenvalues fields 660, which identify the Eigenvalues of the localstructure tensor for each pixel of the input images 610 and 620. The twoEigenvalues λ₁ and λ₂ for each and every pixel may influence theweighting function 670 by discounting the error values in regions withhigh homogeneity (e.g. low λ₁ and λ₂₎ and low edge dominance (e.g. lowλ₁ relative to λ₂).

After computing a weighting field 675 as in region of support weighting670, the weighted error cost function 680 computes a weighted errorfield 690 as described in the weighting function block 516 of FIG. 5.

Referring to FIG. 7, the previous frame 710, current frame 720, theresulting error field 730, and the initial occlusion map 740 may beinput to a multi-factor sigma filter 750 that operates on the initialocclusion map and field 730 and 740 (similar to the well-knowntwo-factor Bilateral Filter as taught in Tomasi et al., “Bilateralfiltering for gray and color images,” International Conference onComputer Vision, (1998) pp. 839-846), but which may include four or moresigmas (factors): an image color distance function (RGB/YUV) based uponthe previous frame 710, current frame 720, a directional distancefunction for 2D motion vectors based on the weighted distance field 730,a magnitude distance function for 2D motion vectors based on theweighted distance field 730, and occlusion as initially marked ininitial occlusion map 740.

The multi-factor sigma filter 750 may be implemented such that whenspatial smoothing is applied, if a pixel location is marked as occluded,its contribution to the filter bank coefficients may be penalizedheavily, which in turn avoids any unwanted distortions at the objectboundaries. Additionally, difference data from dissimilar motion regionsby virtue of their associated motion vector directions or magnitudes maybe penalized if they are not similar. The multi-factor-sigma filter 750differs from the bilateral filters and their derivatives in many ways,since the original proposed filter used only two parameters: spatialradius and image color difference. In one example, a multi-factor sigmafilter 750 can be represented in equation form as in Eq. 11:

${e^{\prime}( {x,y,t} )} = \frac{\begin{matrix}{\sum\limits_{{({i,j})} \in \Omega}{{g( {{i - x},{j - y},t} )}{r( {{I( {i,j,t} )} - {I( {x,y,t} )}} )}{d( {{{mv}_{u}( {i,j,t} )} -} }}} \\{ {{mv}_{u}( {x,y,t} )} ){D( {{{mv}_{v}( {i,j,t} )} - {{mv}_{v}( {x,y,t} )}} )}{o( {i,j,t} )}{e( {x,y,t} )}}\end{matrix}}{\begin{matrix}{\sum\limits_{{({i,j})} \in \Omega}{{g( {{i - x},{j - y},t} )}{r( {{I( {i,j,t} )} - {I( {x,y,t} )}} )}{d( {{{mv}_{u}( {i,j,t} )} -} }}} \\{ {{mv}_{u}( {x,y,t} )} ){D( {{{mv}_{v}( {i,j,t} )} - {{mv}_{v}( {x,y,t} )}} )}{o( {i,j,t} )}}\end{matrix}}$where e( ) is the error field 690 for the image, o( ) represents theinitial occlusion field 685 provided by the warping compensation 510,and e′( ) is the resultant regularized occlusion map 760; and where g( )is the Guassian spatial distance function, as in Eq. 12:

${g( {{i - x},{j - y},t} )} = {\mathbb{e}}^{{- 0.5}{(\frac{{({i - x})}^{2} + {({j - y})}^{2}}{\sigma_{s}})}}$where r( ) of Eq. 11 is the radiosity function, which observes colordifferences and/or luminance values, and r( ) of Eq. 12 is a suitablecolor difference function based on the RGB or YUV values present in animage I, as in Eq. 13:

${r( {{I( {i,j,t} )} - {I( {x,y,t} )}} )} = {\mathbb{e}}^{{- 0.5}{(\frac{{({{{fC}{({I{({i,j,t})}})}} - {{fC}{({I{({x,y,t})}})}}})}^{2}}{\sigma_{i}})}}$Where fC( ) of Eq. 13. may transform the RGB or YUV values to an HSVcolorspace representation in one example, as in Eq. 14-19:

max RGB = max (R, G, B) min RGB = min (R, G, B)chroma = max RGB − min RGB ${H(\;)} = \{ {{\begin{matrix}{{NaN},} & {{{if}\mspace{14mu}{chroma}} = 0} \\{{\frac{( {G - B} )}{chroma}{mod}\; 6},} & {{{if}\mspace{14mu}{\max{RGB}}} = R} \\{{\frac{( {B - R} )}{chroma} + 2},} & {{{if}\mspace{14mu}{\max{RGB}}} = G} \\{\frac{( {R - G} )}{chroma} + 4} & {{{if}\mspace{14mu}{\max{RGB}}} = B}\end{matrix}{V(\;)}} = {{{chroma}{S(\;)}} = \{ \begin{matrix}{0,} & {{{if}\mspace{14mu}{chroma}} = 0} \\{\frac{chroma}{V},} & {otherwise}\end{matrix} }} $and, where function fC( ) measures the color similarity in HSV colorspace, in one example as in Eq. 20:

${{fC}(\mspace{14mu})} = {{a\lbrack \frac{{{atan}\; 2( {{H( {i,j,t} )}\mspace{14mu},{H( {x,y,t} )}}\mspace{14mu} )} + \pi}{2\pi} \rbrack} + {b( {{S( {i,j,t} )} - {S( {x,y,y} )}} )} + {c( {{V( {i,j,t} )} - {V( {x,y,t} )}} )}}$where a, b and c are user supplied weighting values, which by way of anon-limiting example may be 0.5, 0.5 and 1.0, respectively; and, wherefunction d( ) of Eq. 11 measures the motion vector similarity, which mayinclude for example: a simple magnitude difference measurement functionas in Eq. 21:

${d( {{{mv}_{u}( {i,j,t} )} - {{mv}_{u}( {x,y,t} )}} )} = {\mathbb{e}}^{{- 0.5}{(\frac{{({{{mv}_{u}{({i,j,t})}} - {{mv}_{u}{({x,y,t})}}})}^{2}}{\sigma_{u}})}}$where d( ) is a function to measure simple Euclidian distance betweenmotion vectors and where D( ) is a function as in Eq. 22-25, whereby amethod to independently evaluate motion vector direction similarities isprovided:

${\theta_{1}( {x,y,t} )} = \frac{{{atan}\; 2( {{{mv}( {x,y,t} )}_{x},{{mv}( {x,y,t} )}_{y}} )} + \pi}{2\pi}$${\theta_{2}( {x,y,t} )} = \frac{{{atan}\; 2( {{{mv}( {x,y,{t - 1}} )}_{x},{{mv}( {x,y,{t - 1}} )}_{y}} )} + \pi}{2\pi}$

As a measure of the directional difference between motion vectors fromframe-to-frame, motion vectors that point opposite to each other (180degrees opposed) are considered most different, as in Eq. 24. Two suchvectors would have differences in direction (theta) and magnitude(distance, or D):Δθ(x,y,t)=min[abs(θ₂−θ₁),abs(θ₂−1.0−θ₁),abs(θ₂+1.0−θ₁)]

Further, to numerically emphasize the motion vectors' angulardifferences, the difference of angles is transformed to a logarithmicscale. If motion vectors for a particular spatial location changedirection by a great amount, their difference D( ) is computed on alogarithmic weighting scale as in Eq. 25:D(x,y,t)=Δθ_(log)(x,y,t)=1.442695 log(1.0+2.0×Δθ)

Then, the magnitude differences d( ) of Eq. 21 of the respective opticalflow vectors for each spatial location are transformed to a logarithmicscale. This emphasizes large differences as opposed to small ones. Inone example, values for these are as computed as in Eq. 26:

${d( {x,y,t} )} = {\log\{ {1.0 + {4.0 \times {\max\begin{bmatrix}{\sqrt{{{mv}( {x,y,t} )}_{x}^{2} + {{mv}( {x,y,t} )}_{y}^{2}},} \\\sqrt{{{mv}( {x,y,{t - 1}} )}_{x}^{2} + {{mv}( {x,y,{t - 1}} )}_{y}^{2}}\end{bmatrix}}}} \}}$

The regularization of the output occlusion field values o( ) and errorfield values e′( ) of Eq. 11 is not just using the radius considered,but also includes the difference in motion vectors, the image luminanceand the occlusion markings. This excludes the occluded areas from theoperation and will not introduce any distortions due to imperfect motionestimation vectors.

In turn, the error field is now well-conditioned for a simple,consistent thresholding operation, whereby occlusion field pixelscorresponding to error field values below a given threshold are markedas non-occlusion in the final occlusion map O( ), while those greaterare marked affirmatively as occlusions in the final occlusion map O( ).A non-limiting example is provided in Eq. 27:

${O( {x,y,t} )} = \{ \begin{matrix}0 & {{{if}\mspace{14mu}{e( {x,y,t} )}} > {threshold}} \\1 & {{{if}\mspace{14mu}{e( {x,y,t} )}} \leq {threshold}}\end{matrix} $wherein computation of function O( ) results in the final refinedocclusion map, as shown in FIG. 5, where typical thresholds range from0.003 to 0.006, by way of a non-limiting example.

FIG. 8 illustrates a diagrammatic representation of a machine in theexample form of a computer system 800 within which a set ofinstructions, for causing the machine to perform any one or more of themethodologies discussed herein, may be executed. In some examples, themachine may be connected (e.g., networked) to other machines in a LAN,an intranet, an extranet, or the Internet. The machine may operate inthe capacity of a server machine in client-server network environment.The machine may be a personal computer (PC), a set-top box (STB), aserver, a network router, switch or bridge, or any machine capable ofexecuting a set of instructions (sequential or otherwise) that specifyactions to be taken by that machine. Further, while only a singlemachine is illustrated, the term “machine” shall also be taken toinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methodologies discussed herein.

The example computer system 800 includes a processing device (processor)802, a main memory 804 (e.g., read-only memory (ROM), flash memory,dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), astatic memory 806 (e.g., flash memory, static random access memory(SRAM)), and a data storage device 816, which communicate with eachother via a bus 808.

Processor 802 represents one or more general-purpose processing devicessuch as a microprocessor, central processing unit, or the like. Moreparticularly, the processor 802 may be a complex instruction setcomputing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,or a processor implementing other instruction sets or processorsimplementing a combination of instruction sets. The processor 802 mayalso be one or more special-purpose processing devices such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), network processor,or the like. The occlusion map generator 245 shown in FIG. 2 may beexecuted by processor 802 configured to perform the operations and stepsdiscussed herein.

The computer system 800 may further include a network interface device822. The computer system 800 also may include a video display unit 810(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), analphanumeric input device 812 (e.g., a keyboard), a cursor controldevice 814 (e.g., a mouse), and a signal generation device 820 (e.g., aspeaker).

A drive unit 816 may include a computer-readable medium 824 on which isstored one or more sets of instructions (e.g., instructions of theocclusion map generator 245) embodying any one or more of themethodologies or functions described herein. The instructions of theocclusion map generator 245 may also reside, completely or at leastpartially, within the main memory 804 and/or within the processor 802during execution thereof by the computer system 800, the main memory 804and the processor 802 also constituting computer-readable media. Theinstructions of the occlusion map generator 245 may further betransmitted or received over a network via the network interface device822.

While the computer-readable storage medium 824 is shown in an example tobe a single medium, the term “computer-readable storage medium” shouldbe taken to include a single non-transitory medium or multiplenon-transitory media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or more sets ofinstructions. The term “computer-readable storage medium” shall also betaken to include any medium that is capable of storing, encoding orcarrying a set of instructions for execution by the machine and thatcause the machine to perform any one or more of the methodologies of thepresent disclosure. The term “computer-readable storage medium” shallaccordingly be taken to include, but not be limited to, solid-statememories, optical media, and magnetic media.

In the above description, numerous details are set forth. It isapparent, however, to one of ordinary skill in the art having thebenefit of this disclosure, that examples of the disclosure may bepracticed without these specific details. In some instances, well-knownstructures and devices are shown in block diagram form, rather than indetail, in order to avoid obscuring the description.

Some portions of the detailed description are presented in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as “receiving”, “writing”, “maintaining”, or the like, referto the actions and processes of a computer system, or similar electroniccomputing device, that manipulates and translates to a new coordinatesystem the data represented as physical (e.g., electronic) quantitieswithin the computer system's registers and memories into other datasimilarly represented as physical quantities within the computer systemmemories or registers or other such information storage, transmission ordisplay devices.

Examples of the disclosure also relate to an apparatus for performingthe operations herein. This apparatus may be specially constructed forthe required purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. The high-throughput system and method as disclosed hereinthat improves the perceptual quality and/or the transmission or storageefficiency of existing image and video compression or transmissionsystems and methods solves problems in many fields, such as real-timeefficiency for over-the-top video delivery, cost-effective real-timereduction of public radio-access-network congestion when both uploadingand downloading video and image data from mobile devices, increasedreal-time pass-band television delivery capacity, increase of satellitetransponder capacity, reduction of storage costs for content managementsystems and network DVR architectures, and high-throughput treatment ofimages and video at the distribution network core as but a few examples.

Such a computer program may be stored in a computer readable storagemedium, such as, but not limited to, any type of disk including floppydisks, optical disks, CD-ROMs, and magnetic-optical disks, read-onlymemories (ROMs), random access memories (RAMs), EPROMs, EEPROMs,magnetic or optical cards, or any type of media suitable for storingelectronic instructions.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct a more specializedapparatus to perform the required method steps. Example structure for avariety of these systems appears from the description herein. Inaddition, the present disclosure is not described with reference to anyparticular programming language. It will be appreciated that a varietyof programming languages may be used to implement the teachings of thedisclosure as described herein.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other examples will be apparentto those of skill in the art upon reading and understanding the abovedescription. The scope of the disclosure should, therefore, bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. A method for occlusion region detection,comprising: receiving, by a processing device, a first image and asecond image; estimating, by the processing device, a field of motionvectors between the first image and the second image; motioncompensating, by the processing device, the first image toward thesecond image to obtain a motion-compensated image; comparing, by theprocessing device, a plurality of pixel values of the motion-compensatedimage to a plurality of pixels of the first image to estimate an errorfield; inputting, by the processing device, the error field to aweighted error cost function to obtain an initial occlusion map; andregularizing, by the processing device, the initial occlusion map toobtain a regularized occlusion map, wherein regularizing comprisesapplying a multi-sigma filter to the initial occlusion map.
 2. Themethod of claim 1, wherein regularizing further comprises obtaining aregularized error field.
 3. The method of claim 2, further comprisingrepeating said comparing and said regularizing until a value based on atleast one of the regularized occlusion map or the regularized errorfield is below a threshold value.
 4. The method of claim 1, whereinmotion compensating the first image toward the second image comprisesimage warping the field of motion vectors from the first image towardthe second image.
 5. The method of claim 1, wherein the initialocclusion map and the regularized occlusion map are each based on aweighted error cost function.
 6. The method of claim 5, wherein theweighted error cost function is at least one of a sum-of-squaredifferences measure, a locally scaled sum-of-square differences measure,a normalized cross-correlation measure, or a zero-mean normalizedcross-correlation measure.
 7. The method of claim 6, wherein theweighted error cost function is based on a local weighting over a localregion of support.
 8. The method of claim 7, wherein the local weightingover a local region of support is based on an eigensystem analysis ofthe local structure tensor of the motion-compensated image.
 9. Themethod of claim 7, wherein the local weighting over a local region ofsupport is a gradient-energy weighting over the local region of support.10. The method of claim 9, wherein the gradient-energy weighting over alocal region of support is a sum of statistical variance or localcontrast over the local region of support.
 11. The method of claim 1,wherein applying a multi-sigma filter to the initial occlusion mapcomprises applying a 4-factor sigma filter to the initial occlusion map.12. The method of claim 11, wherein input weights for the multi-factorsigma filter comprise an initial coarse occlusion field estimate and,between the first image and the second image one or more of similaritiesof color value or luminance, similarities of circular values of motionvector directions, or similarities of motion vector magnitudes.
 13. Themethod of claim 12, wherein the multi-factor sigma filter incorporatesone or more weights, such as depth, or discontinuities of arange-to-target field.
 14. A system, comprising: a memory; a processingdevice coupled to and having use of the memory, the processing deviceto: receive a first image and a second image; estimate a field of motionvectors between the first image and the second image; motion compensatethe first image toward the second image to obtain a motion-compensatedimage; compare a plurality of pixel values of the motion-compensatedimage to a plurality of pixels of the first image to estimate an errorfield; input the error field to a weighted error cost function to obtainan initial occlusion map; and regularize the initial occlusion map toobtain a regularized occlusion map, wherein regularizing comprisesapplying a multi-sigma filter to the initial occlusion map.
 15. Thesystem of claim 14, wherein regularizing further comprises obtaining aregularized error field.
 16. The system of claim 15, further comprisingrepeating said comparing and said regularizing until a value based on atleast one of the regularized occlusion map or the regularized errorfield is below a threshold value.
 17. A non-transitory computer-readablestorage medium including instructions that, when accessed by aprocessing device, cause the processing device to perform operationscomprising: receiving a first image and a second image; estimating afield of motion vectors between the first image and the second image;motion compensating the first image toward the second image to obtain amotion-compensated image; comparing a plurality of pixel values of themotion-compensated image to a plurality of pixels of the first image toestimate an error field; inputting the error field to a weighted errorcost function to obtain an initial occlusion map; and regularizing theinitial occlusion map to obtain a regularized occlusion map, whereinregularizing comprises applying a multi-sigma filter to the initialocclusion map.
 18. The non-transitory computer-readable storage mediumof claim 17, wherein regularizing further comprises obtaining aregularized error field.
 19. The non-transitory computer-readablestorage medium of claim 18, further comprising repeating said comparingand said regularizing until a value based on at least one of theregularized occlusion map or the regularized error field is below athreshold value.
 20. The non-transitory computer-readable storage mediumof claim 17, wherein the initial occlusion map and the regularizedocclusion map are each based on a weighted error cost function.
 21. Thenon-transitory computer-readable storage medium of claim 20, wherein theweighted error cost function is based on a local weighting over a localregion of support.
 22. The non-transitory computer-readable storagemedium of claim 21, wherein the local weighting over a local region ofsupport is based on an eigensystem analysis of the local structuretensor of the motion-compensated image.
 23. The non-transitorycomputer-readable storage medium of claim 17, wherein applying amulti-sigma filter to the initial occlusion map comprises applying a4-factor sigma filter to the initial occlusion map.
 24. The system ofclaim 14, wherein applying a multi-sigma filter to the initial occlusionmap comprises applying a 4-factor sigma filter to the initial occlusionmap.